Exemplary embodiments of the present invention relate to a method and device for monitoring at least one vehicle passenger, and a method for operating at least one assistance device.
European patent document EP 1 182 089 B2 discloses a method for warning a driver of a vehicle that involves using vehicle sensors to detect a critical situation. The attention of the driver is determined, wherein an image of the driver is captured by a camera device and the image is processed by a processing unit. The attention of the driver is determined from the image, wherein the line of vision of the driver is determined by the processing unit. When the attention of the driver is determined as being high, the display of the warning is dispensed with. In the case of several warnings being displayed simultaneously, a first warning before a critical situation that lies in the line of vision of the driver is only displayed after a second warning that lies outside the line of vision of the driver.
German patent document DE 10 2009 016 936 A1, the applicant of which is the assignee of the present application and the entire contents of which are herein expressly incorporated by reference, discloses a driving assistance system to support a driver of a vehicle in the case of fatigue. The driving assistance system comprises a fatigue detection system to detect fatigue of the driver. In the case of fatigue of the driver detected by means of the fatigue detection system, a flashing warning light of the vehicle is activated to warn other road users.
Furthermore, the article by Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard, “Non-Contact, Automated Cardiac Pulse Measurements using Video Imaging and Blind Source Separation,” Opt. Express 18, 10762-10774 (2010), the complete contents of which are expressly incorporated by reference herein, discloses a method in which, in essence, the heart rate of the person is determined by means of video sequences of faces of people, captured using low resolution video cameras. The differences in brightness of the light reflected by the skin are measured and analyzed. First, the positions of faces in the field of vision of the camera are identified, the video image in this section is broken down into red, green and blue portions and analyzed. During a heartbeat, the blood vessels—above all the arteries—expand slightly, as the pressure increases. This expansion causes an increase of the optical absorption and therefore also a decrease of the intensity of the light that is reflected by the face.
Exemplary embodiments of the present invention provide an improved method and device for monitoring at least one vehicle passenger in a vehicle and an improved method for operating at least one assistance device of a vehicle.
In a method for monitoring at least one vehicle passenger in a vehicle, images of the vehicle passenger are captured by means of at least one image capturing unit and are analyzed by means of an image processing unit.
According to the invention, at least one vital sign of the vehicle passenger is determined by the image analysis of the captured images. This enables a vital state of the vehicle passenger, for example of a driver of the vehicle, to be assessed and, in the case of a detected deterioration of the vital state, suitable measures to be taken, for example functions of an assistance device of the vehicle to be activated, in order at least to indicate a potential risk of accident or, if necessary, to prevent an accident, help to be requested and/or the comfort of the vehicle passenger(s) to be increased, in order to improve the vital state in this way. In this case, an image capturing unit already installed in the vehicle, for example in the region of a steering column, and directed towards the vehicle passenger(s), preferably towards a driver of the vehicle, preferably serves as the image capturing unit. Such image capturing units are, for example, installed to detect the attention of the driver of the vehicle. A video camera with low resolution, for example a so-called CCD camera, is sufficient to carry out the method. Special lighting equipment is also not required; ambient lighting that is normal in the vehicle, for example daylight, is sufficient.
In particular, a pulse rate or heart rate and/or a breathing rate and/or a pumping capacity of the heart and/or a blood pressure and/or an oxygen concentration in the blood of the vehicle passenger and/or autonomous bodily functions, in particular functions of an autonomous nervous system of the vehicle passenger are detected expediently as vital signs. By means of these vital signs, the current vital state of the vehicle passenger can be assessed in a sufficiently precise manner, in order to be able to suggest, for example, if necessary, an inattention of the vehicle passenger, for example of the driver of the vehicle, or a threatening or already occurring unconsciousness or a severe health problem. Advantageously, the detected vital signs are recorded and saved in the vehicle, in such a way that they are available, for example, in an emergency situation in the case of medical care. They can, for example, be transmitted to a hospital or an emergency response centre with an automatically initiated emergency call, in such a way that fast and adequate help can be initiated.
Preferably, the at least one vital sign is determined by detecting an optical absorption value and/or an intensity value of reflected light of a skin region of the vehicle passenger and/or by determining a change of this optical absorption value and/or of this intensity value. During a heartbeat, the blood vessels, in particular the arteries, expand slightly, as the pressure increases. This expansion causes an increase of the optical absorption and therefore also a decrease of the intensity of the reflected light, which is reflected by the skin region. Vital signs, in particular the pulse rate, can be determined by determining the differences in brightness of the light reflected by the skin region over the course of time. This method is also called photoplethysmography. The foundations of the method are described, for example, in the article by Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard, “Non-Contact, Automated Cardiac Pulse Measurements using Video Imaging and Blind Source Separation,” Opt. Express 18, 10762-10774 (2010), the entire contents of which are herein expressly incorporated by reference.
Preferably, at least one region suited to image analysis is looked for in each of the images during the image processing and only this region is analyzed. This region is a region in the images, in which a sufficiently high skin region of the vehicle passenger is depicted. Preferably a region of the images is looked for and analyzed as a region suited to image analysis, which contains the face of the vehicle passenger, as in this region a sufficiently large region of skin that is able to be analyzed is available and as image capturing units, which are, in particular, already installed in the vehicle, the image capturing units being used to detect attention, are directed towards the face of the vehicle passenger, in particular of the driver of the vehicle. With the method, one or several vital signs of several vehicle passengers can be determined at the same time, wherein the image regions are determined, in which the faces of the vehicle passengers are situated and are analyzed separately from one another.
Color images are captured expediently as images of the vehicle passenger, in particular so-called RGB images, wherein the region of the images suited to image analysis is split into color channels and at least one color channel is analyzed to determine the at least one vital sign. For example, the region suited to image analysis is divided into red, green and blue portions, i.e. color channels. Therein, preferably, the green portion or color channel is analyzed, as in this color channel the precision of the achieved results is the highest.
Preferably, a spectral power density of the color channel of the image suited to image analysis is determined by means of a Fast-Fourier Transform. The differences in brightness of the light reflected by the skin region over the course of time and thus the vital sign(s) can then be determined by means of the determined spectral power density.
The vital signs determined in this way can also be combined with values of the vehicle passengers, which have been determined in a different way, in order to thus obtain a better overall picture of a state of the vehicle passenger. In this way tiredness or inattention of the driver of the vehicle can, for example, be detected early on and with a small margin of error. Thus, for example, movements of the respective vehicle passenger, in particular of the driver of the vehicle, can be detected by sensors in a vehicle seat, which can indicate restlessness, stress or the beginnings of tiredness. This can, for example, be combined with the determined pulse rate, in order to obtain a better estimation.
In a device for monitoring at least one vehicle passenger in a vehicle, in particular for carrying out the method for monitoring at least one vehicle passenger in a vehicle, images of the vehicle passenger are able to be captured by means of at least one image capturing unit and able to be analyzed by means of a image processing unit. According to the invention, at least one vital sign of the vehicle passenger is able to be determined by means of the image capturing unit through image analysis of the captured image. This enables a vital sign of the vehicle passenger, for example of a driver of the vehicle, to be assessed and, in the case of a detected deterioration of the vital state, suitable measures to be taken, for example functions of an assistance device of the vehicle to be activated, in order at least to indicate a potential risk of accident or, if necessary, to prevent an accident, help to be requested and/or the comfort of the vehicle passenger(s) to be increased, in order to improve the vital state in this way, as has already been described for the method, which is able to be carried out by means of the device, to monitor at least one vehicle passenger in a vehicle.
In a method for operating at least one assistance device of a vehicle using the method for monitoring at least one vehicle passenger, at least one function of the assistance device of the vehicle is activated if a determined value of a vital sign deviates from a predetermined value or value region for this vital sign. In this way the assistance device can intervene in a supporting manner in the case of poor or deteriorating vital signs of the vehicle passenger(s), in particular of a driver of the vehicle, in order to, for example, indicate a potential risk of accident or, if necessary, prevent an accident, request help and/or increase comfort of the vehicle passenger(s), in order to improve the vital sign in this way.
An intervention is preferably activated in a steering device, braking device and/or in a drive train of the vehicle and/or an optical, audible and/or haptic warning device as a function of the assistance device. In this way the vehicle, as described, for example, in German patent document DE 10 2009 016 936 A1 can be automatically steered to the side of the road by a corresponding intervention in the steering device, braking device and/or the drive train of the vehicle and there brought to a standstill. Particularly advantageously, the intervention in the steering device, braking device and/or in the drive train of the vehicle can occur in such a way that the vehicle can be steered autonomously to the next available emergency assistance installation, for example to the A & E department of the next available hospital.
Alternatively or additionally, the warning devices of the vehicle can be activated, for example external warning devices in the form of a flashing warning system and/or a horn of the vehicle, in order to warn other road users, and or warning devices in an interior space of the vehicle can be activated, in order to warn the vehicle passenger(s), for example in order to wake a sleeping driver of the vehicle or to indicate a poor or deteriorating vital state of the driver of the vehicle to other vehicle passengers, in such a way that these can actuate the steering wheel and/or a handbrake of the vehicle, in order to brake the vehicle and/or to steer the vehicle to the side of the road.
Alternatively or additionally, at least one comfort function and/or emergency call function is activated expediently as a function of the assistance device. By activating the comfort function or a plurality of such comfort functions, the vital signs of the vehicle passenger can be improved, for example if the determined vital parameter(s) indicate fatigue, in particular of the driver. Then, for example, a massage function of a vehicle seat, interior ventilation and/or an air conditioning function of the vehicle can be activated. In this case, essential oils or other refreshing agents can be introduced into the interior of the vehicle to refresh the vehicle passenger. Furthermore, the current vital situation can be indicated to the vehicle passenger in question by optical, audible and/or haptic means, preferably combined with a recommendation of suitable countermeasures, for example to take a refreshment break, move regularly and/or to take on sufficient fluid. The vehicle passenger can, in this way, for example, also be reminded to take required medication. Furthermore, devices of the vehicle, which support and/or calm the driver of the vehicle, can be activated in a targeted manner, for example a navigation device or an audio system of the vehicle, i.e. for example a car radio.
Additionally, the vehicle passenger can be warned of threatening health risks, for example of threatening circulation problems. In this case it is particularly advantageous to determine the vital signs of the vehicle passenger(s), for example of the driver of the vehicle, over a long period of time, for example several days or weeks, and to analyse the course thereof, in such a way that slow deteriorations can be determined. In particular in the case of older vehicle passengers and/or passengers who may be endangered or compromised in terms of health, in particular drivers of the vehicle, the sense of security and well-being thereof is considerably increased by this. In a medical emergency, an automatic emergency call can additionally be made, wherein, preferably, the detected vital signs are transmitted to an emergency call centre or a hospital, in such a way that fast and adequate medical help can be initiated. For this purpose, current positional data of the vehicle is also preferably determined and transferred with the emergency call.